Imaging apparatus and imaging system

ABSTRACT

Provided is an imaging system capable of stereo-photographing with both of visible and infrared images, and improving color reproducibility in visible-light-photographing. The imaging system includes two imaging sensors, and two DBPFs that have transmittance characteristics in a visible light band and a second wavelength band, are respectively provided correspondingly to the two imaging sensors, and serve as optical filters. The imaging system has: at least four kinds of filters, which have mutual different spectral transmission characteristics corresponding to wavelengths in the visible light band and whose transmissions in a second wavelength band approximate each other; and two color filters provided so as respectively correspond to the two imaging sensors. The imaging system measures a distance to a target based on two visible or infrared image signals.

CROSS REFERENCE

This application is a continuation under 35 U.S.C. § 111(a) of U.S.patent application Ser. No. 16/306,863, filed on Dec. 3, 2018, which inturn is the U.S. National Phase under 35 U.S.C. § 371 of InternationalApplication No. PCT/JP2016/066612 filed on Jun. 3, 2016, the entirecontents of each of which applications are hereby incorporated byreference in their entirety.

TECHNICAL FIELD

The present invention relates to an imaging apparatus and an imagingsystem.

BACKGROUND ART

A combination of image recognition technology and biometricauthentication has been recently used to advance development of:surveillance cameras for detecting criminals, shoplifters, andterrorists, etc. by face authentication; and in-vehicle cameras utilizedfor automatic driving of automobiles.

For example, known has been a system in which a surveillance cameradetects a distance by using a stereo camera and detects intrusion ofsuspicious individuals (see Patent Document 1). Patent Document 1discloses a “distance-measurement image recognition apparatus using astereo image(s)” as a method of measuring a shape and distance of anobject. Known has been a technique called a stereo method for obtaininga distance(s) from a stereo image(s). In this stereo method, first, tworight and left images called stereo images are inputted, andcorresponding points of the right and left images (where a target objectat a certain position of the left image is projected in the right image)are obtained by calculating a feature amount of the images. Details ofhow to obtain the corresponding points are described as an “imagematching method” in, for example, Patent Document 2. Obtaining thecorresponding points of the right and left images then makes it possibleto calculate a distance to an object surface by the triangulation'sprinciple, so that the distance to the object and the shape of theobject surface can be known.

Patent Document 1 also proposes a moving-object recognition apparatusthat can detect a moving object(s) with high accuracy and high speed andmeasure its shape and distance by using a known correspondencerelationship with a stereo image(s).

Surveillance cameras and automatic driving cameras are required so as tophotograph regardless of location and time such as outdoor and indoorlocations, and day and night. However, no sufficient illumination may beoften obtained depending on such a situation in photographing. In thiscase, it is conceivable to take an infrared photography by usinginfrared illumination which human beings cannot see. It is conceivableto take an infrared photography by using infrared light as illuminationfor illuminating a distant place in consideration of an influence ofheadlight on an oncoming vehicle(s) at night also in the automaticdriving camera. In any case, it is conceivable to take a visible-lightphotography, without illumination, in the daytime which there is a highpossibility of making an amount of visible light sufficient, and to takethe infrared photography by using the infrared illumination, which it isdifficult for human eyes to catch, in needing night illumination.

Considering such situations, it is preferable that each of thesurveillance camera and the automatic driving camera can simultaneouslyphotograph with the visible light and infrared light.

An imaging apparatus such as a surveillance camera continuouslyphotographing day and night detects infrared light and photographs atnighttime. A photodiode (light receiving element) serving as a lightreceiving portion of an imaging sensor such as a CCD sensor or CMOSsensor can receive light up to a near infrared wavelength band of about1300 nm, so that the imaging apparatus using those imaging sensors makesit possible in principle to photograph up to an infrared band.

Incidentally, a wavelength band of light with high human's visibility is400 nm to 700 nm, so that when the imaging sensor detects near-infraredlight, the human eyes appear reddish on an image(s) detected by thesensor. This makes it desirable to provide, in front of the imagingsensor, an infrared cut filter for blocking light in the infrared bandand to remove light having a wavelength of 700 nm or more in order tomatch sensitivity of the imaging sensor with the visibility of the humanbeings in photographing in the daytime or at an indoor bright place(s).Meanwhile, providing no infrared cut filter is required in photographingat night or in a dark place.

Conventionally known as the above-mentioned imaging apparatus have been:an imaging apparatus that an infrared cut filter is manually attached toor detached from; or an imaging apparatus that an infrared cut filter isautomatically inserted into and removed from. Furthermore, disclosed isan imaging apparatus not requiring the insertion and removal of theabove-described infrared cut filter. For example, proposed is an opticalfilter having; a transmission characteristic in a visible light band; acutoff characteristic in a first wavelength band adjacent to a longwavelength side of the visible light band; and a transmissioncharacteristic in a second wavelength band which is a part of the firstwavelength band (see Patent Document 3). This filter makes it possibleto transmit light in both of the visible light band and a secondwavelength band which is away from the visible light band on a longwavelength side, i.e., an infrared side of the visible light band.

Hereinafter, called a DBPF (double band pass filter) will be the opticalfilter that transmits, as mentioned above, light in the visible lightband and light in the second wavelength band on the infrared side andblocks light in the other wavelength band.

Additionally, recently advanced as biometrics authentication hasdevelopment of various authentication technologies such as fingerprints,faces, irises, veins, signatures, voiceprints, and walking. However,recited as biometrics authentication used with image recognitiontechnologies about the images captured by the above-describedsurveillance camera and automatic driving camera are face authenticationand iris authentication.

RELATED ART DOCUMENTS Patent Documents

Patent Document 1: Japanese Patent Application laid-open No. H3-81878

Patent Document 2: Japanese Patent Application laid-open No. S62-107386

Patent Document 3: Japanese Patent No. 5009395

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

The DBPF of Patent Document 3 does not block, all the time, the light inthe second wavelength band (a comparatively narrow wavelength bandincluded in the infrared wavelength band) included in the infrared (nearinfrared) wavelength band, and causes the light to be transmitted. Thatis, unlike a case of using the infrared cut filter that cuts the longwavelength side of the visible light band, the photography in thevisible light band is subjected to no little influence of infrared lighttransmitting the second wavelength band.

For the photography in the visible light band, a color filter is used inthe imaging sensor that takes a color photography. Color filters, whichcorrespond to respective pixels of the imaging sensor and in whichrespective color regions (filter portions) of red, green, and blue arearranged in a predetermined pattern, basically have a peak oftransmittance of light in each color wavelength band and blocktransmission of light in the other color wavelength bands.

However, the long wavelength side of the visible light band has adifferent light transmittance depending on each color region andwavelength, but basically leads to transmitting of light. Therefore, iftransmitted in the second wavelength band on the infrared side like theabove-described DBPF, the infrared light passes through the color filterand reaches the photodiode (light receiving element) of the imagingsensor, thereby bringing an increase in an amount of electrons generatedby the photoelectric effect due to the photodiode.

Further, in performing both of color photography with visible light andphotography with infrared light illumination, for example, the colorfilter in which the respective color regions of red, green, and blue arearranged in the predetermined pattern(s) is provided with an infraredlight region (infrared region) that has a peak of the lighttransmittance in the above-mentioned second wavelength band. That is, anarrangement (pattern) of the color filters is composed of four regionsof red R, green G, blue B, and infrared IR. In this case, the infraredlight region blocks the light in the visible light band and mainlytransmits the light in the second wavelength band. Therefore, it isconceivable that an infrared light component(s) is removed from an imagesignal of each color of red, green, and blue by using an image signal ofinfrared light outputted from the imaging sensor which receives lightpassing through the infrared light region of the color filter. However,even such a signal processing has made it difficult to reproduce almostthe same color as that in having color-photographed through use of theinfrared cut filter. When right and left signals are made stereo tocalculate a distance, some deviation between right and left signallevels has brought a factor of generating an error(s) in parallaxcalculation.

Furthermore, when face authentication is used by utilizing aphotographed image(s) of a camera, there are roughly two types of faceauthentication; a type in which a user adjusts a user's face on apredetermined camera in an entry/exit management system of an office orbuilding, a boarding procedure at an airport, immigration/registrationmanagement, or the like; and a type of authenticating a unspecifiedlarge number of users that the user has unknowingly photographed by aplurality of cameras at locations such as public facilities, airports,or transportation facilities for the purpose of pursuit of criminals,prevention of terror, and early detection of suspicious individuals,etc. The former type can recognize the face authentication with highprecision even by recent technologies since photographing conditions arelimited. Meanwhile, the latter type greatly influences a recognitionrate since photographing conditions such as illumination condition, facedirection, and angles greatly vary due to an environmental change(s).

In order to perform suspicious-individual detection by usingsurveillance cameras, use of both visible light and infrared light makesit possible to continuously photograph for 24 hours without depending ona photographing location(s) and a photographing time. Further, if thephotographed image can be made as clearly as possible with less noiseand higher resolution, the ability to early detect a suspiciousindividual(s) and to analyze a situation(s) at a time of crime isexpected to be significantly improved.

Additionally, when the photography with infrared light and thephotography with visible light as described above are used incombination, even the automatic driving camera that has a configurationof measuring a distance through a stereo method by using the two camerasobtains a clear image(s) with less noise, thereby being capable ofachieving improvement of accuracy of the image recognition.

From the above, it is desirable that: simultaneous photography of bothof an infrared image and a visible image is possible; a level of imagequality such as noise, resolution, and color reproducibility is equal toor more than that of a visible image including no normal infrared image;and further stereo photography is possible with two cameraconfigurations.

The present invention provides a technique capable of: photographingboth of a visible image(s) and an infrared image(s) and capturing a highquality image(s) by improving color reproducibility at a time ofphotographing with visible light.

Means for Solving the Problems

An imaging apparatus or imaging system according to the presentinvention includes: an imaging element; a filter configured to have atleast a characteristic of transmitting a visible light wavelength rangeand an infrared light wavelength range and to filter a signal from theimaging element based on the characteristic; a signal processorconfigured to process the signal filtered by the filter to output avisible light signal and an infrared light signal; a moving-objectregion extractor configured to generate, from the infrared signaloutputted from the signal processor, information on a moving object inan image photographed by the imaging element; and a signal outputcontroller configured to transmit, outside, a first data containing atleast one of the visible and infrared light signals outputted from thesignal processor, and a second data based on information on the movingobject generated by the moving-object region extractor.

Further, an imaging apparatus or imaging system according to the presentinvention includes: two imaging elements; two filters configured to haveat least a characteristic of transmitting a visible light wavelengthregion and an infrared wavelength region and to filter signals from theimaging elements based on the characteristic; two signal processorsconfigured to process the signals filtered by the filters and to outputa visible light signal and an infrared light signal; a distancecalculator configured to use two visible image signals and/or twoinfrared image signals outputted from the signal processors to calculatea distance to a to-be-photographed subject that has been photographedwith a visible image based on the visible image signals and/or aninfrared image based on the infrared image signals; a moving-objectregion extractor configured to generate, from the infrared signalsoutputted from the signal processors, information on a moving object inan image photographed by the imaging elements; and a signal outputcontroller configured to transmit, outside, a first data that containsat least one of the visible or infrared light signals outputted from thesignal processors, a second data that is based on information on themoving object generated by the moving-object region extractor, and athird data that is based on a distance image generated by the distancecalculator.

Effects of the Invention

The present invention makes it possible to obtain an image(s) with highquality. More specifically, for example, one aspect of the presentinvention makes it possible to simultaneously photograph both ofhigh-quality infrared and visible images with a camera that isconfigured by one imaging sensor and one optical film, and so to improvevisibility even under nighttime or an environmental change such asinsufficient lighting. Further, another aspect of the present inventionmakes it possible to measure more accurately a distance of the objectwith both of the infrared and visible images, and to provide itsinformation to an exterior system together with the visible or infraredimage. Additionally, yet another aspect of the present invention makesit possible to extract a moving object in an image with higher speed byusing not the visible image but the infrared image, and to provide itsinformation to an exterior system together with the visible or infraredimage.

BRIEF DESCRIPTIONS OF THE DRAWINGS

FIG. 1 is a schematic diagram showing an imaging system according toEmbodiment 1 of the present invention;

FIG. 2 is a schematic diagram showing a configuration of an imagingsensor of the imaging system according to Embodiment 1 of the presentinvention;

FIG. 3 is a graph showing transmittance spectra of a DBPF and a colorfilter of the imaging sensor in the imaging system according toEmbodiment 1 of the present invention;

FIG. 4 is a schematic diagram showing one configuration example of thecolor filter of the imaging system according to Embodiment 1 of thepresent invention;

FIG. 5 is a block diagram for explaining a signal processor of theimaging system according to Embodiment 1 of the present invention;

FIG. 6 is a flowchart for explaining a communication flow between acamera and a controller of the imaging system according to Embodiment 1of the present invention;

FIG. 7 is a diagram showing one configuration of various kinds of piecesof image information handled by the imaging system according toEmbodiment 1 of the present invention;

FIG. 8 is a schematic diagram showing an imaging system according toEmbodiment 2 of the present invention;

FIG. 9 is a diagram showing one configuration of various kinds of piecesof analysis metadata information handled by the imaging system accordingto Embodiment 2 of the present invention;

FIG. 10 is a flowchart for explaining a communication flow between acamera and a controller of the imaging system according to Embodiment 2of the present invention;

FIG. 11 is a view showing an example of an image screen of an imagehandled by the imaging system according to Embodiment 2 of the presentinvention;

FIG. 12 is a diagram showing one configuration example of pieces ofanalysis metadata information handled by the imaging system according toEmbodiment 2 of the present invention;

FIG. 13 is a schematic diagram showing an imaging system according toEmbodiment 3 of the present invention;

FIG. 14 is a schematic diagram showing another configuration example ofthe imaging system according to Embodiment 3 of the present invention;

FIG. 15 is a schematic diagram showing another configuration example ofthe imaging system according to Embodiment 3 of the present invention;

FIG. 16 is a schematic diagram showing yet another configuration exampleof the imaging system according to Embodiment 3 of the presentinvention;

FIG. 17 is a diagram showing one configuration of various kinds ofpieces of image information handled by the imaging system according toEmbodiment 3 of the present invention;

FIG. 18 is a diagram showing one configuration of various kinds ofpieces of image information handled by the imaging system according toEmbodiment 3 of the present invention;

FIG. 19 is a diagram showing one configuration of pieces of analysismetadata information handled by the imaging system according toEmbodiment 3 of the present invention;

FIG. 20 is a diagram showing one configuration example of pieces ofanalysis metadata information handled by the imaging system according toEmbodiment 3 of the present invention;

FIG. 21 is a graph showing transmittance spectra of a DBPF and a colorfilter of an imaging sensor in the imaging system according toEmbodiment 3 of the present invention;

FIG. 22 is a diagram showing a processing flow of an imaging apparatusin the imaging system according to Embodiment 3 of the presentinvention;

FIG. 23 is an example of employing the imaging system according toEmbodiment 3 of the present invention; and

FIG. 24 is an example of employing the imaging system according toEmbodiment 3 of the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS Embodiment 1

Hereinafter, an embodiment of the present invention will be describedwith reference to the drawings.

FIG. 1 shows a configuration example of an imaging system according toEmbodiment 1 of the present invention. The imaging system roughlyincludes one or more imaging apparatuses 100((a)-(n)), and one or morecontroller apparatuses 200. The imaging apparatuses 100 and thecontroller apparatuses 200 are connected via a network 303. Although thenetwork 303 is described on the premise of a wired LAN (Local AreaNetwork) in the present embodiment, it may be a general-purpose networksuch as a wireless LAN (WiFi), a USB (Universal Serial Bus), or IEEE1394.

The network 303 uses a standard IP (Internet Protocol) as a networkprotocol, and uses a TCP (Transmission Control Protocol) or UDP (UserDatagram Protocol) as a higher transport protocol. Used for transferringan image(s) photographed by the imaging apparatus 100 is a furtherhigher application protocol, for example, a RTP (Real-time TransportProtocol)/RTCP (RTP Control Protocol), a HTTP (Hyper Text TransferProtocol), or the like. Used for transfer control is a RTSP (Real-TimeStreaming Protocol) or the like. Incidentally, either IPv4 or IPv6 maybe used for the IP. Additionally, utilization of Web services usingtechniques such as HTTP and RTP as described above also makes itpossible to communicate between higher applications. Although not shown,a hub or router may be also interposed for connection of the Internet.

The controller apparatus 200 can control the plurality of imagingapparatuses 100, and can also exchange information with the othercontroller apparatuses 200.

The imaging system of the present embodiment can be used for services orapplications such as surveillance applications and entry/exitmanagement.

The imaging apparatus 100, which has a feature of the presentembodiment, includes a lens 11, an imaging sensor 12, a signal processor13, a signal output controller 14, a communication controller 15, an IF(Interface) 16, an abnormality detector 17, an illuminance monitor 18, aGPS 19, a clock 20, a memory 21, a maintenance IF 22, a controller 23,and an infrared LED 24.

The lens 11 is a photography optical lens, which forms an image at apredetermined focal length on the imaging sensor 12 with visible light301 and infrared light (invisible light) 302 from a subject to bephotographed, and includes a plurality of lenses.

The image sensor 12 is a unit configured to spectrally separate, throughvarious kinds of filters, the visible light and infrared light of theimage formed by the lens 11, photoelectrically convert them, and outputa plurality of pixel signals corresponding to predetermined wavelengthcomponents.

The signal processor 13 is a unit configured to: process an outputsignal(s) outputted from the image sensor 12; perform, to an imagesignal(s), an interior processing, an image processing for removing aninfluence of infrared light passing through the second wavelength bandduring the color photographing, and an image processing such as gammacorrection, white balance, or RGB matrix correction; and output theoutput signals of the visible and infrared images.

A visual image signal and an infrared image signal, which are outputtedfrom the signal output controller 14 and signal processor 13 andobtained by “photographing an object at the same timing”, aretransmitted via the IF 16 to the predetermined controller apparatus 200connected to the network pursuant to an instruction(s) of thecommunication controller 15 or controller 23.

The communication controller 15 is a unit configured to control theimage signal outputted from the signal output controller 14 via the IF16, and transmit and receive a control signal to and from the controllerapparatus 200 via the IF. It is also a unit configured to execute theabove-mentioned network protocol, application protocol, and Web service,etc.

The IF 16 is a communication IF configured to connect the imagingapparatus 100 and the network 303.

The abnormality detector 17 is a unit configured to monitor constantlyor regularly whether any abnormality has occurred in hardware andsoftware of the imaging apparatus 100, and detect such abnormality. Forexample, the abnormality includes a case where the imaging apparatus 100is removed from a predetermined installation place, a case where thephotography of the image(s) is impossible, a case where networkcommunication cannot be made, a case where unauthorized access is made,or the like.

The illuminance monitor 18 is a unit configured to monitor constantly orregularly brightness of a photographing range of the imaging apparatus100 by using an illuminance sensor or the like. When detecting ashortage of illuminance, the illuminance monitor notifies the controller23 of the illumination shortage and irradiates the infrared LED 24.

The GPS 19 is a unit configured to acquire a current position of theimaging apparatus 100 itself from position information received from asatellite. The acquired position information can be also notified to thecontroller apparatus 200 via the IF 16.

The clock 20 is a unit configured to execute current time informationmanagement, and timer setting and release. The time information isautomatically adjusted by using a general-purpose technique such as aNTP (Network Time Protocol) or standard radio wave.

The memory 21 is a storage (ROM (Read-Only Memory) area, FROM (FlashROM) area) configured to store programs, various kinds of pieces ofsetting information, and property information, and is a storage (RAM(Random Access Memory) area) configured to store work data. Here, arecorder may be used by a combination of an external memory (a USBmemory or NAS (Network-Attached Storage)) and a portable medium (amicroflash, an SD card, and a magnetic tape, etc.) besides an built-inmemory.

The maintenance IF 22 is an IF with which a maintenance worker of theimaging apparatus 100 communicates in order to diagnose at times of aupdating processing of the program(s) and occurrence of a failure(s).Further, when the abnormality detector 17 detects any abnormality, themaintenance IF can automatically notify a remote maintenance site of acontent(s) of the abnormality detection.

The controller 23 is a unit configured to control, as a whole,operations of the respective compartments (constituent elements)described above.

Meanwhile, the controller apparatus 200 includes a user IF 201, adisplay 202, a clock 203, a memory 204, a recorder/reproducer 205, acommunication controller 206, an IF 207, a camera manager 208, amoving-object region extractor 209, a face area detector 210, a facefeature point detector 211, a face checker 212, a face DB 213, and acontroller 214.

The user IF 201 is a unit configured to operate the controller apparatus200 through user's use of a remote controller, a touch panel, akeyboard, a mouse, buttons, or the like.

The display 202 is a unit configured to display, on an external orbuilt-in monitor, an operation screen of the controller apparatus 200, avisible or infrared image received via the network 303, a result of theface authentication, and a warning screen, etc.

The clock 203 is a unit configured to execute current time informationmanagement, and timer setting and removal. The time information isautomatically adjusted by using a general-purpose technique such as aNTP or standard radio wave.

The memory 204 is a storage (ROM area, FROM area) for storing programs,various kinds of pieces of setting information, and propertyinformation, and a storage (RAM area) for loading these programs anddata and storing them temporarily, and for storing work data. Here, arecorder may be used by a combination of an external memory (USB memoryor NAS) or a portable medium (microflash, SD card, DVD, Blu-ray(registered trademark) Disc, and magnetic tape, etc.) besides a built-inmemory.

The recorder/reproducer 205 is a unit configured to record andreproduce, in or from the memory 204, the visible and infrared imagesreceived via the network 303 and the IF 207, metadata attached to theseimages, and the like. Those to-be-recorded data are encrypted/decryptedand compressed/expanded as necessary.

The communication controller 206 is a unit configured to transmit andreceives a control signal to and from the imaging apparatus 100 via thenetwork 303 and the IF 207. The communication controller is also a unitconfigured to execute the above-mentioned network protocol, applicationprotocol, and Web service, etc.

The IF 207 is a communication IF configured to connect the controllerapparatus 200 and the network 303.

The camera manager 208 is a unit configured to manage one or moreimaging apparatuses 100 managed by the controller apparatus 200 via thenetwork 303. The camera manager is a unit configured to create, retain,update, and delete information (e.g., IP address, installation location,manufacturer name, model name, introduction time and operation time,function specification, and maintenance contact, etc.) relating to theimaging apparatus 100 to be managed.

The moving-object region extractor 209 is a unit configured to extract amoving object such as a human being, an animal, or an object present inthe visible or infrared image received via the IF 207 or recorded by therecorder/reproducer 205, and acquire its position information. A methodof extracting the moving bodies from the image includes: a method ofcreating a difference image (e.g., a difference image between first andsecond images, and a difference image between second and third images)from a plurality of continuous images (e.g., three) to extract themoving object by their comparison; a method of extracting the movingobject by using a background difference method while a background imageis generated instead of a photographing image; and the like.

The face area detector 210 detects a human-face existing region directlyfrom the visible or infrared image received via the IF 207 or recordedby the recorder/reproducer 205, or from the moving-object regionextracted by the moving-object region extractor 209. A method ofdetecting it includes a technique etc. of a high-speed face detectionalgorithm that uses an integral image(s) of Viola & Johns and a cascadetype discriminator.

The face feature point detector 211 is a unit configured to detectfeature points such as eyes, nose, and mouth ends in the face areadetected by the face area detector 210. This makes it possible to makeimage position correction so as to accurately extract the face features.

The face checker 212 is a unit configured to select optimum features,for identifying individuals, from the feature points detected by theface feature point detector 211, and to perform matching by using theface DB 213. Here, utilized as a feature for distinguishing betweenfaces can be: a method (e.g., a specific method applying principalcomponent analysis) of using the entire grayscale (light and shade)information in the face area; a method of using, as an amount ofcharacteristics, an interval of a local grayscale change and adirectional component (e.g., Elastic Bunch Graph Matching); a methodcombining these methods; and the like. Applied as a matching method cana nearest neighbor method, and a linear discriminant analysis, etc.

In order to be matched by the face checker 212, the face DB 213 is aunit configured to store, in a built-in or external storage medium, datapreviously registering a face image(s). Images that artificiallygenerate an illumination change, and a face direction change, etc. canalso be registered from these registered images. For example, anentry/exit management system registers face images of users who arepermitted to enter a specific area or users who are employees. Themanagement system can also register additionally an image capable ofbeing confirmed as the identical person as a result of the faceauthentication at a specific place. Here, this face DB 213 may be anexternal DB that can be accessed via the network 303 instead of thecontroller apparatus 200. For example, a surveillance camera system inan airport etc. utilizes a face DB of a suspect, a terrorist, or thelike provided by a police or a legal agency. Additionally, the DB may beshared between the plural controller apparatuses.

The controller 214 is a unit configured to control, as a whole,behaviors of the respective components described above. Also, if theuser is not a previously registered person (such as a suspicious person)or matches a suspect as a result of the matching by the face checker212, the above controller automatically reproduces a report based onpredetermined format, informs an administrator(s) or police of thereport, and send it to its contact address.

FIG. 2 shows a configuration example of the imaging sensor 12 in thecamera 100.

The imaging sensor 12 includes a sensor body 2, a color filter 3, acover glass 4, and a DBPF 5.

The sensor body 2 is a CCD (Charge Coupled Device) image sensor, and isa unit configured to place a photodiode as a light receiving element foreach pixel. Instead of the CCD image sensor, a CMOS (Complementary MetalOxide Semiconductor) image sensor may be used.

The color filter 3 is provided to the sensor body 2, and is a unitconfigured to arrange, at a predetermined array, respective areas of red(R), green (G), blue (B), and infrared (IR) for each pixel. FIG. 4 showsvariations of the color filter used in the present embodiment.

The cover glass 4 covers the sensor body 2 and the color filter 3,thereby protecting them.

The DBPF 5 is an optical filter formed on the cover glass 4. The DBPF 5is an optical filter that has: a permeability characteristic in avisible light band; a cutoff characteristic in a first wavelength bandadjacent on a long wavelength side of the visible light band; and apermeability characteristic in a second wavelength band serving as apart of first wavelength band. Incidentally, an arrangement position ofthe DBPF 5 is not limited thereto, and may be provided to, for example,the lens 11.

FIG. 3 shows transmittance spectra of R, G, B and IR filters of thecolor filter 3, where a longitudinal axis and a vertical axis representtransmittance and wavelength, respectively. A wavelength range in agraph includes parts of a visible light band and a near infrared band,and shows a wavelength range of, for example, 300 nm to 1100 nm.

As indicated by the symbol R (double line) of the graph, an R filterportion has the substantially maximum transmittance at a wavelength of600 nm, and its long wavelength side becomes maintained in a state ofhaving the substantially maximum transmittance even if the wavelengthexceeds 1000 nm.

As indicated by the symbol G (dashed line with a wide interval) in thegraph, a G filter portion has: a peak at which the transmittance becomesa local maximum near or at a wavelength of about 540 nm; and a portionat which the transmittance becomes a local minimum near or at awavelength of about 620 nm on its long wavelength side. The G filterportion also has an upward tendency of the transmittance toward the longwavelength side from the portion of the local minimum of thetransmittance, and the transmittance becomes the maximum near or at awavelength of about 850 nm. Consequently, the transmittance on the longwavelength side remains the maximum even if the wavelength exceeds 1000nm.

As indicated by the symbol B (broken line with a narrow interval) in thegraph, a B filter portion has a peak at which the transmittance becomesa local maximum near or at a wavelength of about 460 nm, and its longwavelength side has a portion at which the transmittance becomes a localminimum near or at a wavelength of about 630 nm. Consequently, the longwavelength side has an upward tendency of the transmittance, and thetransmittance becomes the maximum at a wavelength of about 860 nm. Thisleads to a state in which the transmittance on the long wavelength sideremains the maximum even if the wavelength exceeds 1000 nm.

An IR filter portion blocks light on a short wavelength side from awavelength of about 780 nm, blocks light on a long wavelength side froma wavelength of about 1020 nm, and has the maximum transmittance near orat a wavelength of about 820 nm to 920 nm.

The transmittance spectrum of each of the R, G, B and IR filter portionsis not limited to that shown in FIG. 3 and the like. However, the colorfilter 3 generally used at present is expected to show a transmittancespectrum close to this. Incidentally, the reference numeral 1 on thevertical axis indicating the transmittance does not mean a lighttransmission of 100% and means, for example, the maximum transmittanceof the color filter 3.

Here, as illustrated by the symbol DBPF (solid line) in the graph, theDBPF 5 used in the present embodiment has high transmittances in twoband, one being a visible light band illuminated by the DBPF (VR), andthe other being an infrared band (second wavelength band) illustrated bythe DBPF (VR) which is at a position slightly distant from the longwavelength side with respect to the visible light band. Additionally,the DBPF (VR) as a high-transmittance band in the visible light band hasa wavelength band of, for example, about 370 nm to 700 nm. The DBPF (IR)as a second wavelength band having a high transmittance in an infraredside has a band of a wavelength of, for example, about 830 nm to 970 nm.

The present embodiment defines a relationship between the transmittancespectrum of each filter portion of the above-described color filter 3and a transmittance spectrum of the DBPF 5 as described below. That is,the DBPF (IR) serving as the second wavelength band and passing throughthe infrared light of the transmittance spectrum of the DBPF 5 has thesubstantially maximum transmittances at all of the R, G, and B filterportions so that the respective filter portions belong to (is includedin) a wavelength band A shown in FIG. 2 and having almost the sametransmittance and also belong to a wavelength band B which transmitslight having the maximum transmittance of the IR filter portion.

Here, the wavelength band A in which the transmittances of therespective R, G, and B filter portions are the same is a portion havinga transmittance of 10% or less about a transmittance difference betweenthe respective filter portions. Incidentally, on a short wavelength sideof this wavelength band A, the R filter portion has the substantiallymaximum transmittance while the G and B filter portions have the lowtransmittances. A portion having a transmittance difference between therespective R, G, and B filter portions in DBPF 5 corresponds to aportion serving as the minimal transmittance and nearly blocking lightof the DBPF 5 between the DBPF (VR) serving as a portion having the hightransmittance in the visible light band and the DBPF (IR) serving as aportion having the high transmittance in the second wavelength band ofthe infrared light band. That is, the infrared side cuts off the lighttransmission of a portion increasing the transmittance differencebetween the respective R, G, and B filter portions, and the longwavelength side separate from the above portion has the maximumtransmittance of each of the filter portions so that light transmits thewavelength band A having the same transmittance.

From the above, the present embodiment has an area(s) for transmittinglight in not only the visible light band but also the second wavelengthband on the infrared light side in the DBPF 5 used instead of theinfrared light cut filter, and so leads to being subjected to aninfluence of the light passing through the second wavelength band incolor-photographing by the visible light. However, as described above, awavelength band in the second wavelength band does not transmit light,the respective R, G, and B filter portions in the wavelength band beingdifferent in transmittance, while only another wavelength band thereintransmits light, the respective filter portions in the anotherwavelength band having the maximum transmittances and theirtransmittances being the same.

Further, a wavelength band in the second wavelength band of the DBPF 5transmits light, the IR filter portion in the above wavelength bandhaving the maximum transmittance. Therefore, if it is assumed that fourpixels, which are very close to each other and to which the same lightis irradiated, are each provided with the R, G, B, and IR filterportions, the R, G, B, and IR filter portions in the second wavelengthband pass through light similarly to the above and the light, which hasthe same light amount in each filter portion including IR and serves aslight on the infrared side, leads to reaching a photodiode of theimaging sensor body. That is, a light amount of light beams passingthrough the infrared-side second wavelength band among light beamspassing through the respective R, G, and B filters becomes almost thesame as a light amount of light beams passing through the IR filterportion. For the above-mentioned assumption, a difference between eachoutput signal of the above-assumed pixels from the sensor body 2 havingreceived the light transmitting each of the R, G, and B filters, andeach output signal of the above-assumed pixels from the sensor body 2having received the light passing through the IR filter is basically anoutput signal of each visible-light portion of the R, G, and B, the eachvisible-light portion cutting the infrared-side light passing througheach of the R, G, and B filter portions.

Since the color filter 3 actually arranges any one of the R, G, B, andIR filter portions in each pixel of the sensor body 2, there is a highpossibility that light amounts of light beams of respective colorsirradiated to each pixel will be different. For this reason, forexample, a well-known interpolation method is used per pixel to obtainluminance of the color of each pixel, and a difference between each ofthe interpolated R, G, and B luminances of the pixels and theinterpolated IR luminance similarly can be made luminance of each of theR, G, and B. Incidentally, an image processing method of excluding aninfrared light component(s) from each color of the R, G, and Bluminances is not limited thereto, and may use any method as long asbeing a method capable of finally cutting an influence of light passingthrough the second wavelength band from each color of the R, G, and Bluminances. Even in any method, the DBPF 5 cuts a portion other than aportion having a 10% transmittance of each of the R, G, and B filterportions on an infrared side, i.e., a portion other than a portionhaving a predetermined ratio of the transmittance, and so a processingfor removing the influence of the infrared light becomes easy in eachpixel.

As described above, use of the above image sensor 12 makes it possibleto realize the imaging apparatus 100 capable of both of colorphotography and infrared-light photography. It is generally conceivableto color-photograph at normal photography and to infrared-photograph byusing infrared-light illumination, which is difficult for human beingsto recognize, without using visible-light illumination at night. Invarious kinds of surveillance cameras etc., for example, it isconceivable to night-photograph by infrared light utilizinginfrared-light illuminance when the night photography is performedwithout requiring night illumination or at a place(s) desiring no nightillumination. Further, this makes it possible to utilize useapplications to daytime photography and night photography etc. forobserving wild animals.

When infrared-light photography is used as night-photography, a lightamount(s) of infrared light lacks at night similarly to the visiblelight, and so the infrared-light illumination is needed.

Transmittance spectra (A) and (B) of the DBPF 5 shown in FIG. 21 aredetermined in view of transmittance spectra of the respective R, G, Band IR filter portions and an emission spectrum of light forinfrared-light illumination, e.g., a LED of illumination infrared light.

FIG. 21 shows transmittance spectra R, G, B, and IR of the respectivecolor filter portions similar to FIG. 2 , and an emission spectrumIR-light of the LED illumination additionally to a transmittancespectrum DBPF of the DBPF 5.

Similarly to the DBPF shown in FIG. 2 , the second wavelength bandilluminated by DBPF (IR), which is a portion transmitting the infraredlight of the DBPF shown in FIG. 21(A), has the substantially maximumtransmittances at all of the R, G, and B filter portions so as to belongto the wavelength band A shown by FIG. 2 and having almost the sametransmittance at each of the filter portions and belong to thewavelength band B in which IR filter portion having the maximumtransmittance transmits light.

Additionally thereto, an almost entirety of a wavelength band, whichbelongs to both of the above-mentioned wavelength bands A and B and inwhich an emission spectrum of infrared-light illumination has a peak, isset to be included in the wavelength band of the DBPF (IR).Incidentally, when the infrared-light photography is taken not undernight natural light but under infrared-light illumination, the secondwavelength band indicated by the DBPF (IR) does not require being widerthan a peak width of an optical spectrum of the infrared-lightillumination, and when the spectrum of the infrared-light illuminationis included in both of the above-mentioned wavelength bands A and B, apeak portion of the transmittance of the DBPF 5 indicated by the DBPF(IR) may be provided as a second wavelength band so as to have almostthe same peak width as that of a peak whose apex has, for example, about860 of the emission spectrum of the infrared-light illumination.

That is, in FIG. 21A, a peak of an emission spectrum of infrared-lightillumination indicated by IR-light is on a short wavelength side of eachof the above-mentioned wavelength bands A and B, and overlaps a peak ofthe DBPF indicated by DBPF (IR) 2 overlaps a peak of an emissionspectrum of the IR-light on a short wavelength side of each of thewavelength bands A and B that include the second wavelength band of theDBPF indicated by the DBPF (IR).

Also in a graph shown in FIG. 21(B) similarly to FIG. 21(A), theemission spectrum of the infrared-light illumination is added to thegraph of FIG. 2 , and the second wavelength band indicated by the DBPF(IR) and having a portion in which an infrared side of a transmittancespectrum of the DBPF 5 has high transmittance is combined with the peakof the emission spectrum indicated by the IR-light of theabove-mentioned infrared-light illumination.

In FIG. 21(B), illumination in which a peak of an emission spectrum islonger in wavelength than that of FIG. 21(A) is used as infrared-lightillumination. This peak is included in the wavelength bands A and B andexists on a long wavelength side of each of the wavelength bands A andB. Correspondingly thereto, the second wavelength band indicated by theDBPF (IR) of the DBPF 5 is provided so as to overlap the peak indicatedby the IR-light of the infrared illumination in each of theabove-mentioned wavelength bands A and B.

The second wavelength band of the DBPF 5 may be any of the secondwavelength bands shown in FIGS. 2 and 21 , and the second wavelengthband may have only to be included in both of the above-mentionedwavelength bands A and B. Additionally, when the wavelength band thatbecomes the peak of the emission spectrum of the infrared-lightillumination used for nighttime infrared-light photography isdetermined, it is preferable that such a wavelength band is included inboth of the above-mentioned wavelength bands A and B and the secondwavelength band of the DBPF 5 is combined with the peak of the emissionspectrum of the infrared-light illumination.

In such an imaging sensor, the second wavelength band transmitting lighton the infrared side of the DBPF 5 has the maximum transmittance of eachfilter portion on each infrared side of the R, G, B, and IR filterportions, and is included in: the wavelength band A in which thetransmittances of the respective filter portions are the same; and thewavelength band B in which the transmittance of the IR filter portionbecomes the maximum. In other words, on the long wavelength side of thevisible light band, only the R filter portion among the R, G, and Bfilter portions becomes the maximum about transmittance, but the G and Bfilter portions become no maximum about transmittance. Thus, lightpassing through a portion, in which the transmittances of the R, G, andB filter portions are not the same and are different, is cut by the DBPF5.

That is, since each of the R, G, B and IR filter portions is set totransmit light in the second wavelength band on the infrared side, allthe transmittances on the infrared sides of the respective filterportions become the same. If light beams with the same light amount areirradiated to the second wavelength band, light amounts of light beamstransmitted to the R, G, B, and IR filter portions become the same. Thismakes it possible to correct colors based on a signal outputted from apixel corresponding to each of the R, G, and B filter portions, andeasily obtain an image(s) that suppresses an influence due to infraredlight passing through the second wavelength band and having a color(s)in color-photographing.

Causing the second wavelength band to correspond to the peak of theemission spectrum of the infrared-light illumination included in thewavelength bands A and B brings efficient use of the light of theinfrared-light illumination and makes it possible to narrow a width ofthe second wavelength band and reduce an influence of the infrared lightpassing through the second wavelength band in color-photographing.

FIG. 5 is a block diagram showing a signal processing of theabove-mentioned signal processor 12.

Explained will be a processing outline with respect to the signaloutputted from the imaging sensor 12 mounting the color filters shown inFIG. 4 .

Output signals of respective R, G, B and IR pixels are sent torespective interior processing blocks 21 r, 21 g, 21 b, and 21 ir. Eachof the interior processing blocks 21 r, 21 g, 21 b, and 21 ir uses aninterpolation processing, which utilizes a well-known method, to convertR, G, B and IR signals so that image data of respective frames of theabove-described color filter 3 are respectively made: image data 20 rwhose pixels are all represented by red R; image data 20 g whose pixelsare all represented by green G; image data 20 b whose pixels are allrepresented by blue B; and image data 20 ir whose pixels are allrepresented by infrared IR.

Next, in order to remove the influence of the infrared light receivedfrom the above-mentioned second wavelength band, infrared-light removalsignal generation blocks 22 r, 22 g, 22 b, and 22 ir generate, from asignal of the IR, a signal for subtracting from each of the R, G, and Bcolor signal. Subtracted from the respective R, G, and B color signalsare the signals generated by the those infrared-light removal signalgeneration blocks 22 r, 22 g, 22 b, and 22 ir for each of the R, G, andB. In this case, the same pixel as described above makes a processing(s)easy since the signal of the IR has just to be basically removed fromthe respective R, G, and B signals. Since even the same pixels areactually different in sensitivity per pixel of each color due to acharacteristic etc. of the filter portion of each pixel, a signal forsubtracting from each of the R, G, and B signals for each of R, G, and Bimages is created from the IR signal.

Next, regarding each of the R, G, and B signals, the image processingblock 23 performs: a well-known RGB matrix processing for correctingcolors by using a determinant to covert each of the R, G, and B signals;a well-known white balance processing for making output values of therespective R, G, and B signals the same about a portion to be whited inthe image; and a well-known gamma correction processing serving ascorrection for outputting an image(s) to a display etc. Then, aluminance matrix block 24 multiplies each of the R, G, B color signalsby a coefficient to generate a signal of luminance Y. Further,subtraction of the signal of luminance Y from the blue B and red Rsignals makes it possible to calculate color difference signals R-Y andB-Y and output the Y, R-Y, and B-Y signals.

Additionally, the IR signal is basically outputted as an image(s) ofblack and white gradation.

FIG. 6 shows a communication flow for exchanging visible images,infrared images, and control commands between the imaging apparatus 100and the controller apparatus 200 that are shown in FIG. 1 . Here, thecommunication flow may use an original communication protocol, but mayuse, for example, a protocol or the like formulated by ONVIF (OpenNetwork Video Interface Forum) as standard communication of thesurveillance camera.

First, when installed at a predetermined location, connected to thenetwork 303, and then powered on, the imaging apparatus 100 isactivated, and the controller 23 of the imaging apparatus 100 executesan initial setting processing. For example, the initial settingprocessing is mainly activation of hardware and an initial parametersetting processing of software such as loading of a program stored inthe memory 21 and acquisition of a current location of the GPS 19.Incidentally, the imaging apparatus may use PoE (Power Over Ethernet),then use a PoE compatible hub, and be activated at timing connected tothe network 303 (Step 601).

When the necessary initial setting processing is completed, thecontroller 23 of the imaging apparatus 100 sets an IP address to be usedin the communication controller 15 and the IF 16. The IP address is setby using a general-purpose network protocol through a method of settinga static IP address, a PC and pad terminal, etc. being directlyconnected in the method by the maintenance IF 22, or through a method ofautomatically setting an IP address by using a DHCP (Dynamic HostConfiguration Protocol) (Step 602).

After completion of the setting of the IP address, the controller 23 ofthe imaging apparatus 100 instructs the communication controller 15 tonotify the controller apparatus 200 of its own presence. Protocols suchas UPnP (Universal Plug and Play) and WS-Discovery (Web Services DynamicDiscovery) may be used as a method for automatically discovering anapparatus(es) existing on the network. At a time of such a notification,the controller apparatus may also be set so that the notificationincludes manufacturer name and type name of the apparatus itself,installation location, date and time, and the like (Step 603). In thiscase, the installation location may be information initially set inadvance or information acquired from the GPS 19. The installationlocation may also include information on determination of outdoor orindoor location by using the GPS 19 or illuminance monitor 18.

When receiving the notification, a controller 214 of the controllerapparatus 200 acquires the IP address of the imaging apparatus 100, andso can recognize the presence of the imaging apparatus 100. Thecontroller 214 notifies the administrator that a new imaging apparatus100 is connected via a display 202, and waits for an instruction fromthe administrator as to whether the controller itself manages the newimaging apparatus 100. When receiving an instruction from theadministrator via the user IF 201 or when checking the number of imagingapparatuses 100 currently managed and knowing that the checked numberdoes not reach the maximum number, the above controller automaticallyinstructs the communication controller 206 to transmit, to the imagingapparatus 100, an acquisition request of installation functioninformation (Step 605).

The controller 23 of the imaging apparatus 100, which has received theinstallation-function-information acquisition request, acquires thefunction information stored in the memory 21, instructs thecommunication controller 15 to transmit the function information to thecontroller apparatus 200. For example, the function informationincludes: apparatus management information (presence/absence of supportabout a network, a system, and security, and a parameter value),imaging-apparatus performance information (parameter values about imagequality such as backlight correction, brightness, contrast, whitebalance, focus adjustment, or wide dynamic range, and parameter valuesabout media profiles such as resolution, frame rate, and codec type);PTZ (Pan/Tilt/Zoom) function information (definition of coordinatesystems, movable parameters, and preset positions, etc.); analysisfunction information (analysis function to be supported, types of faceauthentication, and format of analysis results, etc.) (Step 606).

Here, FIG. 7 shows one configuration of information on the imagingapparatus 100 according to the present embodiment. The imaging apparatus100 of the present embodiment has a “visible image” and an “infraredimage” as a classification 701 of an output image and has, as an outputmode 702 to be transmitted to the controller apparatus 200, four typesof “outputting only a visible image”, “outputting only an infraredimage”, “automatically switching both images to any one based onillumination and time and outputting it”, and “simultaneously outputtingboth of visible and infrared images”. Further, as access destinationinformation 703 of the visible image, the controller apparatus 200 has:information on the visible image from the imaging apparatus 100; andURI/URL information accessed for acquiring the actual visible image.Similarly, as access destination information 704 of the infrared image,the controller apparatus 200 has: information on the infrared image fromthe imaging apparatus 100; and URI/URL information accessed foracquiring the actual infrared image. Additionally, this configurationincludes codecs, transfer rates, and resolutions, etc. of the visibleimages capable of being outputted as visible image information 705, andsimilarly includes codecs, transfer rates, and resolutions, etc. of theinfrared images capable of being outputted as infrared image information706. This configuration is an example, and may include the otherinformation.

The controller 214 of the controller apparatus 200, which has receivedthe function information of the imaging apparatus 100, notifies theadministrator of content of the function information via the display 202or automatically confirms the content, and adds, as a management target,it to the camera manager 208 when determining to manage the content bythe controller apparatus 200. The camera manager 208 stores all or apart of the function information in the memory 204 and manages them orit. Further, the controller 214 confirms the analysis function and theauthentication function supported by the controller apparatus 200itself, and determines whether to utilize the images of the imagingapparatus 100. Alternatively/additionally, the controller 214 confirmsthe analysis function information supported by the imaging apparatus100, and may determine an authentication method and an analysis methodto be executed when using the imaging apparatus 100 (Step 607).

When determining to utilize the imaging apparatus 100, the controller214 of the controller apparatus 200: instructs the communicationcontroller 206 in order to set what needs to be changed or set out ofthe parameters included in the function information acquired in Step606; and sends an apparatus setting request to the imaging apparatus100. For example, the present embodiment sets “simultaneously outputtingboth of the visible and infrared images” as the output mode 702 (Step608). Here, for example, the output mode 702 may be determined based onthe installation location of the imaging apparatus 100.

The controller 23 of the imaging apparatus 100, which has received theapparatus setting request, checks whether the received setting isexecutable, and returns its execution result to the controller apparatus200 (Step 609).

Next, the controller 214 of the controller apparatus 200 instructs thecommunication controller 206 to send an access-destination-informationacquisition request for actually acquiring a protocol or parameternecessary for acquiring a visible or infrared image (Step 610).

The controller 23 of the imaging apparatus 100, which has received theaccess-destination-information acquisition request, instructs thecommunication controller 15 to return access destination information(e.g., a media type, a port number, transfer protocol, and payloadnumber, etc.) to media including the access destination information 703of the visible image and the access destination information 704 of theinfrared image (Step 611).

The controller 214 of the controller apparatus 200, which has receivedthe access destination information, subsequently sends the imagingapparatus 100 an acquisition request of session information (DESCRIBE)necessary for receiving the images (Step 612).

The controller 23 of the imaging apparatus 100, which has received thesession-information acquisition request, instructs the communicationcontroller 15 to generate the session information described by using aSDP (Session Description Protocol), and sends the generated sessioninformation to the controller apparatus 200 (Step 613).

The controller 214 of the controller apparatus 200, which has receivedthe session information, instructs the communication controller 206 toestablish an RTSP session with the imaging apparatus 100. Here, the RTSPsession is normally separately established for transferring the visibleimage and for transferring the infrared image (Step 614).

After establishing the RTSP session, the controller apparatus 200prepares to receive these images and prepares for face authentication(Step 615), and the imaging apparatus 100 prepares to transmit a visibleor infrared image (Step 616), and sends its result (Step 617).

When confirming that all the preparations are completed, the controller214 of the controller apparatus 200 instructs the communicationcontroller 206 to transmit a streaming start request (PLAY) to theimaging apparatus 100 (Step 618).

The controller 23 of the imaging apparatus 100, which has received thestreaming start request, instructs the signal output controller 14 tooutput the image requested by the controller apparatus 200 in Step 608,and instructs the communication controller 15 to send the imagingapparatus 100 the images outputted by the signal output controller 14through the RTP used on the session established in the Step 612/613(Step 620).

The controller 214 of the controller apparatus 200 also starts receivingthe images (Step 621).

Thereafter, the controller 214 performs RTP transfers of the visible andinfrared images photographed by the imaging apparatus 100 (Steps 621 and622). Here, in order to reduce a processing load on a controllerapparatus side, the communication controller 15 of the imaging apparatus100 may use a marker bit of a RTP header so that a break(s) of the framebecomes clear.

Each time the predetermined number of frames is transferred, thecommunication controller 15 of the imaging apparatus 100 also sends aRTCP transmission report to the controller apparatus 200. The same timestamp, frame number, and packet count, etc. are stored in the report inorder to indicate that the visible and infrared images have beencaptured simultaneously (step 623).

The controller 214 of the controller apparatus 200, which receives thevisible and infrared images from the imaging apparatus 100, performs theface authentication by using the moving-object region extractor 209,face area detector 210, face feature point detector 211, and facechecker 212 while storing these images in the memory 204 via therecorder/reproducer 205. Then, the controller 214 controls interruptionand stop of the streaming as necessary (Step 624).

The above is the basic communication flow between the controllerapparatus 200 and the imaging apparatus 100.

Here, the above-mentioned communication flow uses the RTP communication,but may use HTTP communication or another unique communication method.The visible and infrared images may be transferred not by separatestreams but by superimposition on the same stream (e.g., a common header(including time stamp and sequence number)+a first visible image+a firstinfrared image+ . . . , etc.). Additionally, simultaneous transfers ofthe both images bring an increase of a usage rate of the communicationband, so that the infrared and visible images may be transferred everyframe and every 30 frames, respectively. Also in this case, framesphotographed at the same timing use the same time stamp and frame numberfor the infrared and visible images.

Here, in Step 623, the imaging apparatus 100 sends a transmission reportto the controller apparatus 200. Similarly thereto, however, thecontroller apparatus 200 may send the imaging apparatus 100 a receptionreport including information on packet loss and transfer delay.

Additionally, in order to indicate that the visible and infrared imagesare photographed simultaneously in Step 623, the present embodimentsends the transmission report setting the same time stamp and framenumber, but may adopt, for example, a method of setting, to the samevalue, a time stamp and sequence number of the RTP header to be sent,and a method of setting the same time stamp and frame number to anexpansion header of the RTP header.

The controller 214 of the controller apparatus 200 instructs therecorder/reproducer 205 to store the received visible and infraredimages in the memory 204, uses the moving-object region extractor 209,face area detector 210, face characteristic point detector 211, and facechecker 212 to detect a person(s) included in the image, and performsthe face authentication about whether the person is a suspiciousindividual etc. At this time, the visible and infrared images obtainedby photographing the same object at the same timing can be acquired.Further, addition of the same time stamp and sequence number brings amerit of facilitating synchronization of the both images, so that thereare a method of face-authenticating the both images to improveauthentication accuracy, and a method of normally face-authenticatingany one (e.g., only infrared image) of the both images and utilizing theother image having the same sequence number in comparing and confirmingthe both images (e.g., there are a portion desiring to grasp additionalinformation such as background and color, and a portion desiring toface-authenticating by another image, etc.).

Additionally, the present embodiment sets “simultaneously outputting theboth visible and infrared images” as the output mode 702 in Step 608,but may change, depending on a time or a surrounding environment, theabove setting to settings such as “simultaneously outputting the bothvisible and infrared images” in the daytime and “outputting onlyinfrared image” in the daytime. Alternatively, when performing the faceauthentication simultaneously while receiving any one of the images orwhen desiring to acquire further information of a person who appears asuspicious individual from a result of the matching by the face checker212, the present embodiment may automatically switch one case to theother case so as to receive the both images on the way during each case.

When determining presence of a suspicious individual or a candidate forthe suspicious individual in the image from a result of verification bythe face checker 212, the controller 214 of the controller apparatus 200notifies the administrator via the display 202 or notifies anothercontroller apparatus 200 via the IF 207 to share information thereon,thereby making it possible to trace the suspicious individual among aplurality of imaging apparatuses 100.

Embodiment 2

Next, described will be a configuration of an imaging system accordingto Embodiment 2 of the present invention.

FIG. 8 shows a configuration example of an imaging system according toEmbodiment 2 of the present invention. Incidentally, the imagingapparatus 100 of Embodiment 1 described above and the imagingapparatuses 800 and 810 of the present embodiment can be mixed andinstalled on the same imaging system, and the controller apparatus 200can manage these imaging apparatuses.

The imaging apparatus 800 of the present embodiment has a configurationin which a moving-object region extractor 801 having almost the samefunction as the moving-object region extractor 209 of the controllerapparatus 200 is mounted on the imaging apparatus 100 of Embodiment 1described above. A configuration other than the above configuration hasalmost the same components as those of the imaging apparatus 100.

The controller 23 of the imaging apparatus 800 inputs, to themoving-object region extractor 801, only the infrared image among thevisible and infrared images outputted from the signal processor 13.Reasons for using only the infrared image include: the ability to detectobjects that cannot be detected with the visible image; the fact thatcontrast between a human being and the background is greater in amountthan that of the visible image and is effective for human detection; andthe like.

The moving-object region extractor 801 extracts a moving-objectregion(s) in the image by using the inputted infrared image, and outputsits number and position information. These results are outputted to thecontroller 25 or the signal output controller 14. The results may bealso stored in the memory 22.

As described above, the above-mentioned imaging apparatus 800 alwaysmonitors the moving-object region in the image by using the infraredimage among the visible and infrared images photographed at the sametiming, and can provide, together with the visible or infrared image,the controller apparatus 200 with information on the moving-objectregion extracted with high accuracy. The controller apparatus 200 canacquire information on the moving-object region together with the image,and so can reduce an image processing burden.

Here, in order to reduce an amount of usage of the communication band onthe network, the controller 23 of the imaging apparatus 800: outputs theimage(s) from the signal output controller 14 via the IF 16 only whenthe moving-object region extractor 801 extracts a moving-objectregion(s); and may not output the image from the signal outputcontroller 14 or may lower a frame rate of the image outputted from thesignal output controller 14 when the moving-object region extractor 801can extract no moving-object region.

Additionally, the controller 23 of the imaging apparatus 800 may combinethe moving-object region extracted by the moving-object region extractor801, and the visible and/or infrared images outputted by the signalprocessor 13, and instruct the signal output controller 14 togenerate/process, on such an image, an image surrounding themoving-object region in a rectangle.

Similarly, the imaging apparatus 810 of the present embodiment has aconfiguration in which a face area detector 802 having almost the samefunction as that of the face area detector 210 of the controllerapparatus 200 is mounted on the above-mentioned imaging apparatus 800.The other components have the same configuration as the imagingapparatus 100. A configuration other than the above configuration hasalmost the same components as those of the imaging apparatus 100.

The controller 23 of the imaging apparatus 810 inputs, to themoving-object region extractor 801, only the infrared image out of thevisible and infrared images outputted by the signal processor 13. Themoving-object region extractor 801 extracts a moving-object region(s) inthe image by using the inputted infrared image, and outputs its numberand position information to the controller 23 or signal outputcontroller 14 and simultaneously inputs them to the face area detector802. The face area detector 802 detects an area where a human face(s)exists from the inputted moving-object region, and outputs the detectedarea to the controller 23 or signal output controller 14.

As described above, the imaging apparatus 810 can: always monitor themoving-object region in the image by using the infrared image out of thevisible and infrared images photographed at the same timing; extract themoving-object region with high accuracy; further detect the area wherethe human face exists from the moving-object region; and provide,together with the visible or infrared image, the controller apparatus200 with the information of the moving-object region and the informationof the face area. The controller apparatus 200 can acquire these piecesof information together with the image, and so reduce the imageprocessing burden.

Here, in order to reduce an amount of usage of the communication band onthe network, the controller 23 of the imaging apparatus 810: outputs animage(s) via the IF 16 from the signal output controller 14 only whenthe face area detector 802 detects a face area(s) of a person(s); andmay not output the image from the signal output controller 14 or mayreduce a frame rate of the image outputted from the signal outputcontroller 14 when the face area detector 802 can extract themoving-object region, but cannot detect the face area of the person.Similarly, in order to detect only an object, the controller 23 of theimaging apparatus 810 may output an image from the signal outputcontroller 14 via the IF 16 only when the face area detector 802 detectsthe moving-object region in which the face area of the person cannot bedetected.

Further, the controller 23 of the imaging apparatus 810 may compare theface area extracted by the face area detector 802 (and the moving-objectregion extracted by the moving-object region extractor 801) and thevisible image and/or infrared images outputted by the signal processor13, and instruct the signal output controller 14 to generate/process, onsuch an image, an image surrounding the moving-object region in arectangle.

The communication flow between each of the imaging apparatuses 800 and810 and the controller apparatus 200 is substantially the same as thecontent described in FIG. 6 of Embodiment 1, so that only a differencetherebetween will be described below.

First, in Step 606 of FIG. 6 , function information with which theimaging apparatuses 800 and 810 provide the controller apparatus 200 isadded, as an example, to the information shown in FIG. 7 , and theinformation as shown in FIG. 9 is provided as the above-describedanalysis function information. That is, the function informationcontains: content indicating that the imaging apparatus 800 itselfinstalls a “moving-object region extraction function”; and contentsindicating that the imaging apparatus 801 installs a “moving-objectregion extraction function” and a “face area detection function”.

In the present embodiment, these pieces of information are set asanalyzed metadata, and contain: a classification 901 of the analyzedmetadata shown in FIG. 9 and servicing as function information (analysisfunction information); an output mode 902 of the analyzed metadata;access destination information 903 of moving-object-region metadata;access destination information 904 of face-area metadata; accessdestination information 905 of moving-object-region/face-area metadata;information 906 of moving-object-region metadata; and information 907 offace-area metadata.

In Step 607, the controller 214 of the controller apparatus 200, whichhas received the function information, confirms an analysis function andan authentication function supported by the controller apparatus 200itself, and determines whether to use the analyzed metadata outputtedfrom the imaging apparatus 800. This makes it possible to select, forexample, use of only “position information of moving-object region” ofthe analyzed metadata for both of the imaging apparatuses 800 and 810,or use of only the “position information of face area” of the analyzedmetadata in the imaging apparatus 810 without using the analyzedmetadata of the imaging apparatus 800.

FIG. 10 shows a communication flow for transmitting a visible image, aninfrared image, and analyzed metadata between each of the imagingapparatus 800 and 810 and the controller apparatus 200. In thisexplanation, it is assumed that the imaging apparatus 800 transmitsanalyzed metadata of the “moving-object-region position information” andthe imaging apparatus 810 transmits both of analyzed metadata of the“moving-object-region position information” and analyzed metadata of“face-area position information”. The imaging apparatuses 800 and 810also establish a session for transferring the analyzed metadataadditionally to the visible and infrared images in Step 614 of FIG. 6 .

The imaging apparatuses 800 and 810 start frame transfers of the visibleand infrared images (Steps 1001 and 1002). Each time the predeterminednumber of frames is transferred (Steps 1003 and 1004), the imagingapparatuses 800 and 810 transmit the analyzed metadata extracted by themoving-object region extractor 801 and the face area detector 802 (Step1005). Here, the analyzed metadata may be sent at the timing when themoving-object region or face area is detected.

The controller 214 of the controller apparatus 200, which has receivedthe analyzed metadata 1200, checks whether the analyzed metadata 1200includes information on a moving-object region(s) or information on aface area(s) (Step 1006). Then, if the information on the moving-objectregion is not included, the controller 214 uses its own moving-objectregion extractor 209 to perform an extraction processing of themoving-object region (Step 1007).

Meanwhile, when information of the moving-object region or face area isincluded, the controller 214 confirms whether information on the facearea is included (Step 1008). Then, if the information on the face areais not included (that is, only information on the moving-object regionis included), the controller 214 uses the received information on themoving-object region and its own face area detector 210 to perform adetection processing of the face area (Step 1008).

On the other hand, when information on the face area is included, thecontroller 214 uses the received information on the face area and itsown face-feature point detector 211 to extract a face feature point(s)(Step 1010), and uses the face checker 212 to perform matching therewith(Step 10100).

FIG. 11 shows image pictures handled by the imaging apparatuses 800 and810. An image 1100 is an example of a visible image photographed by theimaging apparatuses 800 and 810. An image 1101 is obtained by removing abackground from the image 1100, thereby extracting only themoving-object region. In this image example, three areas (portions eachsurrounded by a broken-line square) of (A), (B), and (C) can beextracted. An image 1102 is obtained by further extracting a face areafrom the image 1101. In this example image, two areas (portions eachsurrounded by a solid square) of (a) and (b) can be extracted.

FIG. 12 shows a configuration example of analyzed metadata 1200 sent bythe imaging apparatuses 800 and 810 in Step 1005.

The analyzed metadata 1200 is roughly composed of a communication header1201 and a payload 1210. The communication header 1201 is, for example,similar to the RTP header, the HTTP header, and the like.

The analyzed metadata is stored in the payload 1210. For example, thepayload is configured by: a frame number 1211 of the infrared image usedfor extracting the moving-object region or face area; a frame number1212 of the visible image; the maximum number 1213 of the moving-objectregions extractable by the imaging apparatuses 800 and 810; amoving-object-region extraction number 1214 (n in this case) actuallyextracted by the moving-object region extractor 801; coordinateinformation 1 to n (1215 to 1216) of the extracted moving-object region;a face-area extraction number 1218 (m≤n in this case) actually extractedby the face-area detector 802; and coordinate information 1 to m (1219to 1220) of the extracted moving-object region.

As described above, in addition to the visible and infrared images, theimaging apparatuses 800 and 810 of the present embodiment can provide,simultaneously with a necessary image output(s), the controllerapparatus 200 with information on moving-object regions and/orinformation on human areas that have been accurately extracted by usingthe infrared images.

Meanwhile, the controller apparatus 200 can omit a conventionalprocedure(s) by using the received information of the moving-objectregion and human area immediately, thereby making it possible to reducean execution time of the face authentication shorter than a conventionalexecution time. This is effective in reducing the processing load of thecontroller apparatus 200 when many imaging apparatuses are managed byone controller apparatus 200.

Here, the present embodiment describes an example in which the imagingapparatuses 800 and 810 transmit, to the controller apparatus 200, atleast any one of the visible and infrared images and the analysisparameters. However, in order to reduce an amount of data on thenetwork, the present embodiment may transmit the analysis parameters andan image(s) of only a portion(s) (moving-object region and face area)indicated by the analysis parameters.

Additionally, when the moving-object region in the image is firstdetected by the moving-object region extractor 801, the controller 23 ofthe imaging apparatuses 800 and 810 may: hold a frame number of thecorresponding image; track a target of the moving-object region fromimages photographed sequentially to the above image until the targetdoes not exist; and add the above frame number as attribute informationof the coordinate information in the analyzed metadata 1200 shown inFIG. 12 . This makes it possible to easily grasp the frame numberincluded in the moving-object region and calculate a time since thecontroller apparatus 200 refers to the analyzed metadata 1200.

Embodiment 3

Next, described will be a configuration of an imaging system accordingto Embodiment 3 of the present invention.

The imaging apparatuses according to Embodiments 1 and 2 described abovehave photographed the visual and infrared images by using one set oflens 11, an imaging sensor 12, and a signal processor 13. An imagingapparatus of the present embodiment has a configuration in which twosets of lenses 11, an imaging sensor 12, and a signal processor 13 arearranged on each of right and left sides, thereby making it possible totake stereo images (distance images) composed of two right and leftimages by each of visible light and infrared light.

FIG. 13 shows a configuration example of the imaging system of thepresent embodiment. This imaging system includes one or more imagingapparatuses 1300 and a controller apparatus 1310.

As described above, the imaging apparatus 1300 includes the two sets oflenses 11, the imaging sensor 12, and the signal processor 13, and newlyincludes a correction parameter calculator 1301 and a distancecalculator 1302. The two lenses 11(a) and 11(b) are arranged on rightand left sides so that their optical axes are parallel to each other. Aconfiguration other than the above configuration basically has almostthe same components as those of the imaging apparatuses 100, 800, and810 of Embodiments 1 and 2.

The correction parameter calculator 1301 sets a parameter such as acorrection value (e.g., a correction value added to, subtracted from,multiplied by, or divided by signals such as a visible and infraredimage signals, an infrared signal, and each color signal) of a cliplevel or a signal level so as to approximate signal strengths (signallevels) of respective visible images outputted from the two signalprocessors 13(a) and 13(b) so that two visible image signals (twoinfrared image signal) approximate about their signal levels. Correctionamounts of the image signal correction processor 203 are each set inview of outputs from the two signal processors 13(a) and 13(b), so thatlevels of the image signals are matched. A processing of matching thelevels of the right and left image signals can be performed to both ofthe infrared and visible image signals.

That is, the correction parameter calculator 1301 determines thecorrection amount based on the signal levels of the image signalsoutputted from the two signal processors 13(a) and 13(b) so that thesignal levels of the image signals outputted from the two signalprocessors 13(a) and 13(b) are approximated. Consequently, for example,even if luminance levels of two pieces of image data are different,different portions in a subject to be photographed are recognized as thesame portion (corresponding points), which makes it possible to suppressan error(s) occurring about a distance to be measured, and occurrence ofits error.

The distance calculator 1302 calculates a distance to an object by usingthe two visible or infrared image signals respectively inputted from thetwo signal processors 13(a) and 13(b). At this time, the distancecalculator 1302 determines the same to-be-photographed subject(corresponding point) from the two images, and detects a parallax(disparity) as a positional difference of the same to-be-photographedsubject onto the image, thereby obtaining the distance similarly to aconventional technique. That is, the corresponding points for measuringthe parallax are determined by image recognition, and the distance iscalculated based on a parallax that is a difference between positions ofthe corresponding points in the image. Then, a stereo image (distanceimage) is generated based on the distance information corresponding toeach pixel, and is outputted to the signal output controller 14.

The signal output controller 14 can provide the controller apparatus1310 with the stereo image (distance image) generated by theabove-mentioned distance calculator 1302 additionally to the two visibleand infrared images photographed on the right and left sides.

As described above, the present imaging apparatus 1300 makes it possibleto simultaneously acquire the visible and infrared images of theto-be-photographed subject, and calculate the distance from both of theimages. At this time, matching a position of the visible image with thatof the infrared image makes it possible to prevent the distance measuredbetween the both images from varying.

Here, the above-mentioned distance calculator 1302 uses two visibleimages and two infrared images to calculate the respective distances,and then generates two stereo images (distance images) to output them asthey are. Alternatively, the distance calculator 1302 may: compare thetwo generated stereo images to output a distance image of any one stereoimage if a difference between their pieces of distance information iswithin a threshold value and to output distance images of both stereoimages if the difference therebetween exceeds the threshold value;output a distance image(s) (e.g., a distance image calculated with aninfrared image being prioritized, and a distance image showing a valueclose to a distance, etc.) previously set for output; or separatelyoutput, as analyzed metadata, an area portion exceeding the thresholdvalue. FIG. 22 shows an example of a processing outline of the distancecalculator 1302.

In accordance with an instruction from the controller apparatus 200, thecontroller 23 of the present imaging apparatus 1300 uses the signaloutput controller 14 to control an image(s), which is outputted via theIF 16, among the visible and infrared images outputted from the twosignal processors 13(a) and 13(b) and the stereo images (distanceimages) outputted from the distance calculator 1302. For example, thefollowing use becomes possible: when the present imaging apparatus 1300is installed in a place (e.g., a toilet or changing room) requiringprivate protection, only the stereo image is outputted; and when theimaging apparatus 1300 is installed in a place requiring high security,all the images are outputted.

Meanwhile, in order to use the stereo image (distance image)additionally to the visible and infrared images or use only the stereoimages to perform the analysis and authentication processings, thecontroller apparatus 1310 mounts a different moving-object regionextractor 1311, face area detector 1312, face feature point detector1313, face checker 1314, and 3D face DB 1315 instead of themoving-object region extractor 209, face area detector 210, face featurepoint detector 211, face checker 212, and face DB 213 of the controllerapparatus 200 according to Embodiments 1 and 2. This makes it possibleto, for example, acquire three-dimensional data on irregularities(concave and convex) of the face in performing the face recognition, andaccurately and easily detect the face area and the face feature point byusing the acquired data.

Further, the controller apparatus 1310 acquires the stereo image(distance image) from the imaging apparatus 1300, thereby referring tothe distance of the moving-object region extracted by the moving-objectregion extractor 209 and making it possible to judge whether to performthe face authentication if the distance is within a predetermineddistance or to perform no face authentication if not.

FIG. 23 shows examples for displaying states of the imaging apparatuses1300(a), 1300(b), and 1300(c) whose installation locations are differentfrom that of the controller apparatus 1310. The controller apparatus1310 uses the visible or infrared image and the distance image that arereceived from the imaging apparatus 1300(a) installed at an entrance ofan office, building, or the like, performs 3D-based face authentication,and displays its authentication result(s). This facilitates confirmationof visitors and suspicious individuals, which is useful for solvingcongestion at a reception(s). Further, the controller apparatus 1310uses the distance image received from the imaging apparatus 1300(b)installed in a shop of a public facility, commercial facility, or thelike, and displays information on the number of people viewing amerchandise shelf(s) or information (e.g., sex, height, face direction,body direction, and attitude, etc.) on a range in which the personcannot be specified. This is useful for judgment of a degree of interestetc. from constituency, a line of sight, or an attitude of a shopper,and for sales capabilities enhancement/marketing about merchandise ordisplay. The controller apparatus 1310: uses the visible, infrared, anddistance images received from the imaging apparatus 1300(c) installed onan outdoor place of an amusement park, a park, or the like; extracts aperson(s) from the images, performs a 3D-based face authentication; anddisplays, when the person is confirmed as a previously registeredperson, the confirmed person with a distance image (s) and only thenot-confirmed person with a visible or infrared image(s). Alternatively,the controller apparatus 1310 displays information (e.g., sex, height,facial direction, child accompanying state, and attitude, etc.) on arange in which a person cannot be specified. This is useful for safetysecurity of visitors and early detection of suspicious person.

Next, FIG. 14 shows another configuration example of the imaging systemof the present embodiment. This imaging system is configured by one ormore imaging apparatuses 1400 and a controller apparatus 1410. Theimaging apparatus 100 of Embodiment 1, the imaging apparatuses 800 and810 of Embodiment 2, and the above imaging apparatus 1300 may be mixedon the network 303 although not shown. The controller apparatus 1410 canmanage all of the above imaging apparatuses.

The imaging apparatus 1400 mounts two moving-object region extractors1401(a) and 1401(b) on the imaging apparatus 1300. The moving-objectregion extractor 1401 may be similar to the moving-object regionextractor 1311 of the above-mentioned controller apparatus 1310. Aconfiguration other than the above configuration has almost the samecomponents as those of the imaging apparatus 1300.

The moving-object region extractors 1401(a) and 1401(b) are unitsconfigured to use the infrared images outputted from the two signalprocessors 13(a) and 13(b) to extract a moving-object region from eachof the images. Information on these extracted moving-object regions canbe outputted to the signal output controller 14 or the controller 23 andprovided to the controller apparatus 1410 similarly to Embodiment 2described above. The moving-object region extractors 1401(a) and 1401(b)also use the stereo images (distance images) outputted from the distancecalculator 1302 to extract moving-object regions, and can extract themoving-object regions with high accuracy by comparing their results withthe above method. Alternatively, the moving-object region extractors mayfirstly use stereo images (distance images) to extract moving-objectregions, and then use infrared images to confirm in more detail only apart(s) of the extracted moving-object regions.

Here, the controller 23 can also refer to the information on the twomoving-object regions outputted from the moving-object region extractors1401(a) and 1401(b), compare a number(s) and position(s) extracted, andtransmit their comparison results as analyzed metadata. The controllerapparatus 1410 can utilize the analyzed metadata and make a selectionabout which of the right and left visible or infrared images are usedfor face authentication. For example, regarding results of theextraction of the moving-object regions by the moving-object regionextractors 1401(a) and 1401(b), if the result of the extraction by themoving-object region extractor 1401(a) (or moving-object regionextractor 1401(b)) is larger in number of moving-object regions, thecontroller 23 of the imaging apparatus 1400 sends information on themoving-object region extracted by the moving-object region extractor1401(a) (or moving-object region extractor 1401(b)), and the visible orinfrared image outputted from the signal processor 13(a) (or signalprocessor 13(b)).

Meanwhile, the controller apparatus 1410 mounts, on the controllerapparatus 1310, the face area detector 210, face feature point detector211, face checker 212, and face DB 213 of the controller apparatus 200described in Embodiment 1, and a synthetic judgment unit 1411.

This makes it possible for the imaging apparatus 1400 to combine andperform: a face authentication processing (using the visible andinfrared images), which uses the face area detector 210, face featurepoint detector 211, face checker 212, and face DB 213 described inEmbodiment 1; and a face authentication processing (using a visible,infrared, and stereo images) that uses the face area detector 1312, facefeature point detector 1313, face checker 1314, and 3D face DB 1315 asdescribed above. The synthetic determination processor 1411 is a unitconfigured to perform final judgement of a person authenticationresult(s) based on results of performing both of the face authenticationprocessings. Performing two different kinds of face authenticationmethods as described above makes it possible to perform the faceauthentication with higher accuracy.

Similarly, FIG. 15 shows yet another configuration example of theimaging system of the present embodiment. This imaging system isconfigured by one or more imaging apparatuses 1500 and a controllerapparatus 1510. The imaging apparatus 100 of Embodiment 1, imagingapparatuses 800 and 810 of Embodiment 2, and imaging apparatuses 1300and 1400 may be mixed on the network 303 although not shown. Thecontroller apparatus 1510 can manage all of the above imagingapparatuses.

The imaging apparatus 1500 mounts two face area detectors 1502(a) and1502(b) on the imaging apparatus 1400. This face area detector 1502 maybe similar to the face area detector 1312 of the controller apparatus1310. A configuration other than the above configuration has the samecomponents as those of the imaging apparatus 1400.

The face area detectors 1501(a) and 1501(b) are units configured to useinformation on the moving-object regions outputted from the twomoving-object region extractors 1401(a) and 1401(b) to extract a facearea of a person. Information on these extracted face areas can beoutputted to the signal output controller 14 or controller 23, andprovided to the controller apparatus 1510 similarly to Embodiment 2described above.

Here, the controller 23 can: refer to the information on the two faceareas outputted from the face area detectors 1501(a) and 1501(b);compare a number(s) and position(s) extracted and then a facedirection(s); and transmit their compared results as analyzed metadata.The controller apparatus 1510 uses the analyzed metadata, and selects animage(s) more suitable for face authentication, thereby making itpossible to perform the face authentication with higher accuracy.

Meanwhile, the controller apparatus 1510 newly mounts, on the controllerapparatus 200 or controller apparatus 1410, an authentication methodselector 1511, iris detector 1512, iris checker 1513, and iris DB 1514.

The authentication method selector 1511 is a unit configured to use thevisible, infrared, and stereo images (distance image) and the analysisparameter information, etc. received from the imaging apparatus 1500 tomake a selection about which of the face authentication and irisauthentication should be performed. For example, the authenticationmethod selector performs the iris authentication if an object fallswithin a predetermined distance range, and performs the faceauthentication if otherwise. Alternatively, the authentication methodselector normally performs the face authentication, and further performsthe iris authentication when complying with a condition(s) capable ofthe iris authentication.

The iris detector 1512 uses the infrared image received from the imagingapparatus 1500 and the analysis parameter including the face areaextracted from the image to detect iris positions of human eyes, furtherdetect a boundary between an iris and an white eye and a boundarybetween the iris and a pupil, determine (identify) an iris area(s), andgenerate a pupil cord(s). Incidentally, applied to those methods may beany of well-known methods.

Based on the information detected by the iris detector 1512, the irischecker 1513 uses the iris DB 1514 to perform matching similarly to theface authentication.

This makes it possible for the controller apparatus 1510 to use thevisible, infrared, and stereo images (distance image) received from theimaging apparatus 1500, and the analysis parameter information, etc. toselect optimum biometric authentication, which makes it possible toperform personal authentication with higher accuracy.

FIG. 24 shows an example in which the controller apparatuses 1410 and1510 process photographed images of the imaging apparatus 1500 installedat an airport, a building entrance, or the like. In this example, thecontroller apparatus 1410 obtains the face area and distance informationas the visible images and analysis parameters from the imaging apparatus1500, thereby performing the 2D face authentication, whose imageprocessing load is comparatively light, to the long-distance face area,or performing the 3D face authentication, whose image processing load iscomparatively heavy, to the short-distance face area. Further, thecontroller apparatus 1510 acquires the face area and distanceinformation as visible and infrared images and analysis parameters fromthe imaging apparatus 1500, thereby using the visible image with respectto the long-distance face area to perform the face authentication, orusing the infrared image with respect to the short-distance face area toperform the iris authentication. Additionally, in order to solvecongestion due to turn waiting, the controller apparatus 1510 can judgea person(s) present at a distance close to a predetermined position inthe photographed image, and also perform the face authentication in thatorder.

FIG. 16 shows another configuration example of the imaging systemaccording to the present embodiment. This imaging system shows, forexample, an example mounted on a portable terminal such as a smartphoneor tablet.

FIGS. 17 and 18 show configuration examples of function information andanalysis parameters relating to the imaging apparatuses 1400 and 1500used in this embodiment.

As shown in FIG. 17 , a stereo image (distance image) generated by theimaging apparatus can be provided to the controller apparatus by atransfer method similar to that of the visible or infrared image.

Alternatively, as shown in FIG. 18 , the above-mentioned stereo imagecan be also provided by a method that is added to the positioninformation of the moving-object region and the position information ofthe face area. In this case, only distance information on coordinateareas corresponding to the moving-object region and face area is cut outand added.

FIG. 20 shows a configuration example of sending distance information asa part of analysis parameters. For example, distance information isstored in a payload 1210; the distance information on the extractedmoving-object region (n in number) is stored immediately after thecoordinate information of the moving-object region (2001 and 2002); andthe distance information on the face area (m in number) is storedimmediately after the coordinate information of the face area (2003 and2004). Besides the above configuration, such a configuration may beadopted that the coordinate information and the distance information arealternately stored.

FIG. 19 shows yet another configuration example of the imaging system ofthe present embodiment. An imaging apparatus 1900 of the imaging systemhas one moving-object region extractor 1901, and is configured toextract a moving-object region by using the infrared image of any one ofthe right and left signal processors 13(a) and 13(b). Alternatively, theimaging apparatus 1900 is configured to extract a moving-object regionby using the infrared image of any one of the signal processors 13(a)and 13(b) and the stereo image outputted from the distance calculator1302.

EXPLANATION OF NUMERALS

-   -   2 . . . Sensor Body; 3 . . . Color Filter; 5 . . . DBPF (Optical        Filter); 11 . . . Lens (Optical System); 12 . . . Imaging        Sensor; 13 . . . Signal Processor; 14 . . . Signal Output        Controller; 15 . . . Communication Controller; 16 . . . IF; 23 .        . . Controller; 100, 800, 810, 1300, 1400, 1500 . . . Imaging        Apparatus; 200, 1310, 1410, 1510 . . . Controller Apparatus;        801, 1401 . . . Moving-Objection Region Extractor; 802, 1501 . .        . Face Area Detector; 1301 . . . Correction Parameter        Calculator; 1302 . . . Distance Calculator; and 1600 . . .        Portable Terminal.

The invention claimed is:
 1. An imaging apparatus comprising: a firstimaging element and a second imaging element each comprising multiplepixels arranged in an area for receiving light; a first single filterprovided to the first imaging element, the first single filter beingconfigured to have at least 1) a characteristic of transmitting avisible light wavelength region, 2) a characteristic of blocking a firstlight wavelength region, except for a second light wavelength region, ofa longer wavelength side than the visible light wavelength region, and3) a characteristic of transmitting the second light wavelength regionwhich is part of the first light wavelength region, the first singlefilter being configured to filter light incident on an entirety of thearea of the first imaging element based on the characteristics; a secondsingle filer provided to the second imaging element, the second singlefilter being configured to have at least 1) the characteristic oftransmitting the visible light wavelength region, 2) the characteristicof blocking the first light wavelength region, except for the secondlight wavelength region, of the longer wavelength side than the visiblelight wavelength region, and 3) the characteristic of transmitting thesecond light wavelength region which is part of the first lightwavelength region, the second single filter being configured to filterlight incident on an entirety of the area of the second imaging elementbased on the characteristics; a first signal processor configured toprocess a signal obtained by photographing light that has passed throughthe first single filter and the first imaging element, and output afirst visible light signal and a first signal corresponding to thesecond light wavelength region; a second signal processor configured toprocess a signal obtained by photographing light that has passed throughthe second single filter and the second imaging element, and output asecond visible light signal and a second signal corresponding to thesecond light wavelength region; a distance calculator configured tocalculate a distance to a subject using the first visible light signaloutputted by the first signal processor and the second visible signaloutputted by the second signal processor or use the first signalcorresponding to the second light wavelength region outputted by thefirst signal processor and the second signal corresponding to the secondlight wavelength region outputted by the second signal processor; and asignal output controller configured to add fifth data to first andsecond data or to third and fourth data, or multiply the first andsecond data or the third and fourth data by the fifth data to transmitthe added or multiplied first and fifth data and the added or multipliedsecond and fifth data or the added or multiplied third and fifth dataoutside, the first data being based on the first visible light signaloutputted from the first signal processor to the signal outputcontroller, the second data being based on the second visible signaloutputted from the second signal processor to the signal outputcontroller, the third data being based on the first signal correspondingto the second wavelength region outputted from the first signalprocessor to the signal output controller, the fourth data being basedon the second signal corresponding to the second wavelength regionoutputted from the second signal processor to the signal outputcontroller, wherein switching between a use of the first data and thesecond data and a use of the third data and fourth data in the signaloutput controller is automatically performed depending on time orsurrounding environment.
 2. An imaging apparatus comprising: a firstimaging element and a second imaging element each comprising multiplepixels arranged in an area for receiving light; a first single filterprovided to the first imaging element, the first single filter beingconfigured to have at least 1) a characteristic of transmitting avisible light wavelength region, 2) a characteristic of blocking a firstlight wavelength region, except for a second light wavelength region, ofa longer wavelength side than the visible light wavelength region, and3) a characteristic of transmitting the second light wavelength regionwhich is part of the first light wavelength region, the first singlefilter being configured to filter light incident on an entirety of anarea of the first imaging element based on the characteristics; a secondsingle filter provided to the second imaging element, the second singlefilter configured to have at least the characteristic of transmittingthe visible light wavelength region, the characteristic of blocking thefirst light wavelength region, except for the second light wavelengthregion, of the longer wavelength side than the visible light wavelengthregion, and the characteristic of transmitting the second lightwavelength region which is part of the first light wavelength region,the second single filter being configured to filter light incident on anentirety of the area of the second imaging element based on thecharacteristics; first signal processor configured to process a signalobtained by photographing light that has passed through the first singlefilter and the first imaging element, and output a first visible lightsignal and a first signal corresponding to the second light wavelengthregion; a second signal processor configured to process a signalobtained by photographing light that has passed through the secondsingle filter and the second imaging element, and output a secondvisible light signal and a second signal corresponding to the secondlight wavelength region; a distance calculator configured to calculate adistance to a subject using the first visible light signal outputted bythe first signal processor and the second visible signal outputted bythe second signal processor or use the first signal corresponding to thesecond light wavelength region outputted by the first signal processorand the second signal corresponding to the second light wavelengthregion outputted by the second signal processor; and a signal outputcontroller configured to add fifth data to first and second data or tothird and fourth data, or multiply the first and second data or thethird and fourth data by the fifth data to transmit the added ormultiplied first and fifth data and second and fifth data or the addedor multiplied third and fifth data outside, the first data being basedon the first visible light signal outputted from the first signalprocessor to the signal output controller, the second data being basedon the second visible signal outputted from the second signal processorto the signal output controller, the third data being based on thesecond signal corresponding to the second wavelength region outputtedfrom the second signal processor to the signal output controller, thefifth data being based on a distance image generated by the distancecalculator, wherein the first and second data are transferred at a firsttransfer rate, and the third and fourth data are transferred at a secondtransfer rate.